Systems And Methods For Spatial Thumbnails And Companion Maps For Media Objects

ABSTRACT

Under one aspect, a computer-implemented method includes obtaining one or more media objects having location-related content in response to a search query, wherein the location-related content represents one or more locations corresponding to one or more spatial references in the one or more media objects, obtaining respective spatial thumbnails for the one or more spatial references, obtaining respective summaries of the one or more media objects, obtaining a respective one or more hyperlinks referencing to the one or more media objects, and causing to display for the one or more media objects the respective spatial thumbnails, the respective summaries, and the respective one or more hyperlinks.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of application No. 15,628,097 filedJun. 20, 2017, which is a continuation of application Ser. No.15/226,155 filed Aug. 2, 2016, which is a continuation of applicationSer. No. 11/705,368 filed Feb. 12, 2007, which claims the benefit ofU.S. Provisional Patent Application No. 60/772,088, filed Feb. 10, 2006,the entire contents of which are incorporated herein by reference.

This application is related to U.S. patent application Ser. No.11/705,269, filed Feb. 12, 2007 and entitled “Systems and Methods forSpatial Thumbnails and Companion Maps for Media Objects, the entirecontents of which are incorporated herein by reference.

TECHNICAL FIELD

This application relates to media objects, and more particularly todisplaying information about media objects.

BACKGROUND

Media objects are physical or electronic recordings or representationsof information, thoughts, or emotions. People have created and usedmedia objects for thousands of years. Recently, technologicaladvancements have enabled a great proliferation of media objects and anincrease in exchange of media objects amongst people. This exchange ofmedia objects is frequently facilitated by summaries of media objects.For example, a movie may be recorded in digital form and sold over theInternet to people who wish to view the movie. Typically, theprospective movie buyer examines many possible movie choices and selectsonly a small number. This selection process determines the commercialsuccess of the moviemakers. To increase the number of people who choosetheir movie, the moviemakers typically create “trailers” or shortsequences of excerpted footage from the film to tempt people to get theentire movie. These trailers condense the much larger media object intoa new media object that summarizes the larger media object. The creationof the summary can be as much an art form as the creation of theoriginal media object.

The movie trailer concept has analogs in many forms of media objects.Without the concept of a summary, search and selection of media objectswould be greatly impaired. For example, the back cover or dust jacket ofa book often describes the story line, exciting attributes about theauthor, and praise from other people for the book's value. As anotherexample, search engines generate short textual excerpts from the webpages and other documents that match a user's search request. Theseexcerpts summarize the contents of the document. Image and video searchengines similarly endeavor to produce summaries of the content of themedia objects sought by their users.

Often, a summary is implemented in the same kind of medium as the mediaobject that it describes, e.g. the text of a book is often summarized bytext on the dust jacket, and a miniature clay figurine may summarize thegestalt of a large stone sculpture. However, this is not always true ofsummaries, e.g. a talk show host might verbally summarize the story lineof a book, which is a textual media object. A recording of the talk showhost would be a media object that summarizes the book.

A defining attribute of a summary is that it contains less than all thecontent of the media object. However, summaries sometimes introduce newinformation that is not present in the original media object. Forexample, a talk show host might render opinions of a book, or a movietrailer might arrange its excerpts in an order that communicates aparticular idea that is not obviously present in the movie itself

Generally, a summary is clearly associated with the media object that itsummarizes. This association is often achieved by presenting the summaryvisually adjacent to a means of accessing the media object that itsummarizes. For example, a summary generated by a search engine isusually presented near a hyperlink to the media object that the searchengine is providing in response to a user's search query. As anotherexample, the summary in a dust jacket is physically bound around thebook it summarizes.

With the advent of automated analysis techniques that glean informationfrom digital media using computer algorithms, some summaries arepartially or totally generated without the aide of a human. A now-famoustask in the engineering discipline of natural language processing is“automatic summary generation.” Automatic summary generation is usuallythought of as a text analysis process, however the resampling andresealing of an image to generate a smaller version of an image isanother type of automatic summarization. Such summary images are oftencalled “image thumbnails” because in comparison to the larger image, thesummary image could be as small as the nail material on the end of ahuman's thumb. While possibly hyperbolic, this gives a sense of thescaling down and condensing of information involved in summarization.

Summaries also exist in cartography. Making a more general map of aplace on Earth from information provided by a detailed map has been acentral activity of cartographers for centuries. The act of creating amap at lower scale than an original more detailed map requires carefuljudgment about which features to remove and which to include. Typically,this is called “cartographic generalization” in the sense that thedetailed specific features of a high-scale map must be made more generaland less specific for display at a lower scale. Frequently,large-format, high-scale maps are presented with a small-format,low-scale summary map or “overview” map showing the position of thelarge-format, high-scale map within a larger geographic area that maybemore easily recognized by viewers than the smaller physical areadescribed by the large-format, higher scale map. These overview maps areusually much smaller format than the main map for which they provide anoverview. These maps offer both a summary and more information byshowing surrounding areas of the world. Such overview maps are anexample of a summary that is the same general type of media object asthe media object that it summarizes.

Geographic maps have been used in conjunction with other media objectsfor hundreds of years. For example, a map showing the positions ofevents or other geographic features can be presented along with textdescribing the events or locations. Historians or journalists describinga time period or set of issues in a place often make such narrative mapsby annotating a map information, such as text and images, relating tolocations within the map.

With the advent of computers, digital media objects have taken on manynew forms.

SUMMARY OF THE INVENTION

The present invention provides a new type of summary that presents auser with new information about a digital media object, in a medium thatcan differ from that of the media object that it summarizes. We callthis new type of summary a “spatial thumbnail.” It is a “thumbnail” inthe sense that it gives either a literal or mental image that is like an“image thumbnail” of a larger image. It is “spatial” in the sense thatit depicts location-related content or spatial relationships conveyed inor related to the media object that it summarizes.

One example of a spatial thumbnail is a literal image of a map withvisual indicators showing the positions of locations referenced in adocument along with a piece of text excerpted from the document. Forspecificity, we call this a “spatial thumbnail image for a textdocument” because they have a literal image and are associated with atext document. The detailed description describes systems and methodsfor generating and using these specific types of spatial thumbnails andalso several other types of spatial thumbnails. In general, a spatialthumbnail can be thought of as a media object that summarizes anothermedia object's location-related content.

Under one aspect, a computer-implemented method of displayinginformation about a media object, the media object being associated withlocation-related content corresponding to a location, includes receivinga request to display a representation of the media object; in responseto the request, obtaining a summary of the media object; in response tothe request, obtaining a spatial thumbnail of the media object, thespatial thumbnail including an image of a spatial domain encompassingthe location; and simultaneously displaying both the summary of themedia object and the spatial thumbnail on a visual display.

One or more embodiments include one or more of the following features.The spatial thumbnail further includes a visual indicator indicating thelocation within the image of the domain. The visual indicator representsa subset of the location-related content. The visual indicator includesa marker icon or arrow. Selecting a type of visual indicator based on atype of location reference expressed in the location-related content.The image includes at least one of a two-dimensional image, athree-dimensional image, an animated image, a computer-generatedhologram, and a sculpture. The spatial domain includes one of ageographic domain, a physical and non-geographic domain, and a fictionaldomain. Displaying the summary of the media object includes displayingat least one of a fragment of the media object and an image of the mediaobject. Obtaining the spatial thumbnail of the media object includes oneof obtaining the spatial thumbnail from a repository and generating thespatial thumbnail, wherein generating the spatial thumbnail includesobtaining the location-related content of the media object and obtainingthe image of the spatial domain encompassing the location. Receiving therequest to display the representation of the media object includesreceiving user input referencing the media object. The location-relatedcontent represents at least one of a geographical reference and aspatial relationship within the media object. The location-relatedcontent is generated by an automated geoparsing engine. Thelocation-related content expresses a subset of geospatial referenceswithin the media object, the subset of geospatial references beingselected by the automated geoparsing engine. The media object includesone of a digital document, living organism, paper document, textdocument, rock, video, email message, web page, slide show presentation,spread sheet, equation rendering, music, or book. The summary of thebook includes at least one of a sub-media object excerpted from thebook, a review of the book, an image of a page of the book, and an imagefrom the book. The spatial thumbnail includes a marker representing aspatial reference within the at least one sub-media object excerptedfrom the book. Also displaying a reference to the media object. Thereference includes at least one of a hyperlink to the media object and abibliographic reference to the media object. Also obtaining anddisplaying a plurality of spatial thumbnails for the media object, atleast one spatial thumbnail representing different location-relatedcontent than at least one other spatial thumbnail. Also automaticallygenerating the location-related content associated with the media objectby performing the steps of identifying a spatial reference within themedia object; associating a location with the identified spatialreference, the location being represented by a set of coordinates of aselected coordinate system; and encoding the set of coordinates in thelocation-related content.

Under another aspect, an interface program stored on a computer-readablemedium for causing a computer system with a display device to displayinformation about a media object, the media object being associated withlocation-related content corresponding to a location, causes thecomputer system to perform the functions of receiving a request todisplay a representation of the media object; in response to therequest, obtaining a summary of the media object; in response to therequest, obtaining a spatial thumbnail of the media object, the spatialthumbnail including an image of a spatial domain encompassing thelocation; and simultaneously displaying both the summary of the mediaobject and the spatial thumbnail on a visual display.

Some embodiments include a parsing engine capable of generating thelocation-related content for the media object, the parsing engineincluding instructions for performing the following functions:identifying a spatial reference within the media object; associating alocation with the identified spatial reference, the location beingrepresented by a coordinate of a selected coordinate system; andencoding the coordinates in the location-related content.

Under another aspect, a computer-implemented method of displayinginformation about search results representing media objects, wherein atleast one media object is associated with location-related contentcorresponding to a location, includes: receiving a query from the user;transmitting the query to a search engine; receiving search results fromthe search engine based on the query, each search result representing amedia object; obtaining a spatial thumbnail for a search resultrepresenting a media object that is associated with location-relatedcontent corresponding to a location, the spatial thumbnail including animage of a spatial domain encompassing the location; and simultaneouslydisplaying both the search results and the obtained spatial thumbnail ona visual display.

One or more embodiments include one or more of the following features.At least some of the search results represent digital media objectsstored in a document repository. At least some of the search resultsrepresent digital media objects accessible via the World Wide Web. Alsodisplaying a summary of the represented media object. Displaying thesummary includes displaying at least one of a fragment of therepresented media object and an image of the represented media object.The query from the user includes one of a free-text query, a free-textquery with other query constraints, a selection from a list of options,and navigating through a sequence of links. The spatial thumbnailfurther includes a visual indicator indicating the location within theimage of the domain. The visual indicator represents a subset of thelocation-related content. The visual indicator includes a marker icon orarrow. Selecting a type of visual indicator based on a type of locationreference expressed in the location-related content. The representedmedia object includes one of a digital document, living organism, paperdocument, text document, rock, video, email message, web page, slideshow presentation, spread sheet, equation rendering, music, or book. Thespatial domain includes one of a geographic domain, a physical andnon-geographic domain, and a fictional domain. The spatial domainencompasses location-related content from multiple media objectsrepresented by the search results. The image includes at least one of atwo-dimensional image, a three-dimensional image, an animated image, acomputer-generated hologram, and a sculpture. Obtaining the spatialthumbnail of the represented media object includes one of obtaining thespatial thumbnail from a repository and generating the spatialthumbnail, wherein generating the spatial thumbnail includes obtainingthe location-related content of the represented media object andobtaining the image of the spatial domain encompassing the location. Thelocation-related content includes at least one of a geographicalreference and a spatial relationship within the represented mediaobject. The location-related content is generated by an automatedgeoparsing engine. The location-related content expresses a subset ofgeospatial refererences within the represented media object, the subsetof geospatial references being selected by the automated geoparsingengine. The search results include at least one of a hyperlink and abibliographic reference to the represented media object. Alsoautomatically generating the location-related content associated withthe represented media object by performing the steps of: identifying aspatial reference within the represented media object; associating alocation with the identified spatial reference, the location beingrepresented by a set of coordinates of a selected coordinate system; andencoding the set of coordinates in the location-related content.

Under another aspect, an interface program stored on a computer-readablemedium for causing a computer system with a display device to displayinformation about search results representing media objects, wherein atleast some of the media objects are associated with location-relatedcontent corresponding to locations, causes the computer system toperform the functions of: receiving a query from the user; transmittingthe query to a search engine; receiving search results from the searchengine based on the query, each search result representing a mediaobject; obtaining a spatial thumbnail for a search result representing amedia object that is associated with location-related contentcorresponding to a location, the spatial thumbnail including an image ofa spatial domain encompassing the location; and simultaneouslydisplaying both the search results and the obtained spatial thumbnailson a visual display.

Definitions

For clarity, we define several terms of art:

A “media object” is any physical or electronic object that can beinterpreted as containing information, thoughts, or emotions. Thus, amedia object is a broad class of things, including such diverse objectsas digital documents, living organisms, paper documents, text documents,rocks, videos, email messages, web pages, slide show presentations,spreadsheets, renderings of equations, and music.

A “digital media object” is a media object constructed from binaryelectronic signals or similar computing-machine oriented signals.Frequently, media objects can be stored in digital form, and thisdigital form can be replicated and transmitted to different computersystems many separate times.

A “document” is a media object containing information composed by humansfor the purpose of transmission or archiving for other humans.

A “digital document” is a document that is a digital media object, suchas a file stored in a file system or web server or digital documentrepository.

A “text document” is a document containing character symbols that humanscan interpret as signifying meaning. A “digital text document” is a textdocument that is also a digital document. Typically, digital textdocuments contain character symbols in standardized character sets thatmany computer systems can interpret and render visually to users.Digital text documents may also contain other pieces of informationbesides text, such as images, graphs, numbers, binary data, and othersignals. Some digital documents contain images of text, and a digitalrepresentation of the text may be separated from the digital documentcontaining the images of text.

A “summary” is a media object that contains information about some othermedia object. By definition, a summary does not contain all of theinformation of the other media object, and it can contain additionalinformation that is not obviously present in the other media object.

An “integrated summary” is a set of summaries about the same mediaobject. For example, a web site about a book typically has severalsummaries organized in different ways and in different mediums, althoughthey are all about the same book. An integrated summary can include bothsub-media objects excerpted from the media object summarized by theintegrated summary, and also summary media objects.

To “summarize” is to provide information in the form of a media objectthat is a selection of less than all of the information in a secondmedia object possibly with the addition of information not contained inthe second media object. A summary may simply be one or more excerpts ofa subset of the media object itself. For example, a text search engineoften generates textual summaries by combining a set of excerpted textfrom a document. A summary may be one or more sub-strings of a textdocument connected together into a human-readable string with ellipsesand visual highlighting added to assist users reading the summary. Forexample, a query for “cars” might cause the search engine to provide asearch result listing containing a list item with the textual summary “. . . highway accidents often involve <b>cars<b> that . . . dangerouspileups involving more than 20 <b>cars<b> . . . .” In this example, theoriginal media object contained the strings “highway accidents ofteninvolve cars that” and “dangerous pileups involving more than 20 cars”,and the summary creation process added the strings “ . . . ” and “<b>”and “<b>” to make it easier for users to read the concatenated strings.These substrings from a document and represented to a user are anexample of a “fragment” of a media object.

A “sub-media object” is a media object that is part of a second mediaobject. For example, a chapter in a book is a sub-media object of thebook, and a paragraph in that chapter is a sub-media object of thechapter. A pixel in a digital image is a sub-media object of the digitalimage. A sub-media object is any fragment of a larger media object. Forexample, a fragment of a document might be an image of a portion of thedocument, such is commonly done with digital scans of paper documents. Afragment of a text document might be a string of symbols contained inthe text document and represented to a user. Since digital media objectscan be replicated ad infinitum, a sub-media object of a digital mediaobject can accurately reproduce any portion of the original media objectwithout necessarily becoming a sub-summary.

A “sub-summary” is summary of a sub-media object. A summary may simplybe a set of one or more sub-media objects excerpted from the originalmedia object. The word “sub-summary” is defined here for clarity: asummary of a sub-media object is just as much a summary as other typesof summaries, however in relation to a “containing summary” about alarger fragment of the original work, a sub-summary describes a smallerpart than the containing summary that summarizes the larger fragment.

A “metric space” is a mathematical conceptual entity defined as follows:a metric space is a set of elements possibly infinite in number and afunction that maps any two elements to the real numbers with thefollowing properties. A metric on a set X is a function (called thedistance function or simply distance)

-   -   d:X X X→R

(where R is the set of real numbers). For all x, y, z in X, thisfunction is required to satisfy the following conditions:

-   -   1. d(x, y)≥0 (non-negativity)    -   2. d(x, y)=0 if and only if x=y (identity of indiscernibles)    -   3. d(x, y)=d(y, x) (symmetry)    -   4. d(x, z)≤d(x, y)+d(y, z) (subadditivity/triangle inequality).

A “vector space” is a mathematical conceptual entity with the followingproperties: Let F be a field (such as the real numbers or complexnumbers), whose elements will be called scalars. A vector space over thefield F is a set V together with two binary operations:

vector addition: V×V→V denoted v+w, where v, wϵV, and scalar

multiplication: F×V→V denoted a v, where aϵF and vϵV,

satisfying the axioms below. Four require vector addition to be anAbelian group, and two are distributive laws.

-   -   1. Vector addition is associative: For all u, v, wϵV, we have        u+(v+w)=(u+v)+w.    -   2. Vector addition is commutative: For all v, wϵV, we have        v+w=w+v.    -   3. Vector addition has an identity element: There exists an        element 0ϵV, called the zero vector, such that v+0=v for all        vϵV.    -   4. Vector addition has an inverse element: For all vϵV, there        exists an element wϵV, called the additive inverse of v, such        that v+w=0.    -   5. Distributivity holds for scalar multiplication over vector        addition: For all aϵF and v, wϵV, we have a (v+w)=a v+a w.    -   6. Distributivity holds for scalar multiplication over field        addition: For all a, bϵF and v V, we have (a+b) v=a v+b v.    -   7. Scalar multiplication is compatible with multiplication in        the field of scalars: For all a, bϵF and vϵV, we have a (b        v)=(ab) v.    -   8. Scalar multiplication has an identity element: For all vϵV,        we have 1 v=v, where 1 denotes the multiplicative identity in F.

Formally, these are the axioms for a module, so a vector space may beconcisely described as a module over a field.

A “metric vector space” is a mathematical conceptual entity with theproperties of both a vector space and a metric space.

The “dimension” of a vector space is the number of vectors in theequivalence class of basis vectors that minimally span the vector space.

A “line segment” is a geometric entity in a metric space defined by twoentities in the metric space. These two entities are referred to as the“ends” of the line segment. The line segment is the two ends plus theconcept of a shortest path connecting them, where the path length isdetermined by the metric on the metric space.

A “domain” is an arbitrary subset of a metric space. Examples of domainsinclude a line segment in a metric space, a polygon in a metric vectorspace, and a non-connected set of points and polygons in a metric vectorspace.

A “sub-domain” is a domain which is a subset of another domain. Forexample, if one is considering a domain that is a polygon, then anexample of a sub-domain of that domain is a line segment or subset ofline segments selected from the set of line segments that make up thepolygon.

A “polyline” is an ordered set of entities in a metric space. Eachadjacent pair of entities in the list is said to be “connected” by aline segment.

A “polygon” is a polyline with the additional property that itimplicitly includes a line segment between the last element in the listand first element in the list.

A “polyhedron” is a set of polygons with some of the line segmentsinherent in the underlying polylines are associated with line segmentsfrom other polygons in the set. A “closed” polyhedron is a polyhedron ina metric vector space and every line segment is associated with asufficient number of other line segments in the set that one canidentify an interior domain and an exterior domain such that any linesegment connecting an element of the interior domain to an element ofthe exterior domain is guaranteed to intersect a polygon in the set.

A “spatial domain” is a domain in a metric vector space.

A “coordinate system” is any means of referring to locations within aspatial domain. For example, a so-called Cartesian coordinate system ona real-valued metric vector space is a tuple of real numbers measuringdistances along a chosen set of basis vectors that span the space. Manyexamples of coordinate systems exist. “Unprojected latitude-longitude”coordinates on a planet, like Earth, are an example of two-dimensionalspherical coordinates on a sphere embedded in three-dimensional space. A“datum” is a set of reference points from which distances are measuredin a specified coordinate system. For example, the World Grid System1984 (WGS84) is commonly used because the Global Position System (GPS)uses WGS84 as the defining datum for the coordinates that it provides.For coordinate systems used to describe geographic domains, one oftenspeaks of “projected” coordinate systems, which are coordinates that canbe related to unprojected latitude-longitude via mathematical functionsand procedures called “projection functions.” Other types of coordinatesystems use grids to divide a particular domain into subdomains, e.g.the Military Grid Reference System (MGRS) divides the Earth intosubdomains labeled with letters and numbers. Natural language referencesto places are a coordinate system in the general sense that people oftenrecognize a phrase like “Cambridge” as meaning a place, but there may bemany such places. Such ambiguity is typically not tolerated in thedesign of coordinate systems, so an important part of constructinglocation-related content is coping with such ambiguity, either byremoving it or describing it or simply stating that it exists.

A “physical domain” is a spatial domain that has a one-to-one and ontoassociation with locations in the physical world in which people couldexist. For example, a physical domain could be a subset of points withina vector space that describes the positions of objects in a building. Anexample of a spatial domain that is not a physical domain is a subset ofpoints within a vector space that describes the positions of genes alonga strand of DNA that is frequently observed in a particular species.Such an abstract spatial domain can be described by a map image using adistance metric that counts the DNA base pairs between the genes. Anabstract space, humans could not exist in this space, so it is not aphysical domain.

A “geographic domain” is a physical domain associated with the planetEarth. For example, a map image of the London subway system depicts ageographic domain, and a CAD diagram of wall outlets in a building onEarth is a geographic domain. Traditional geographic map images, such asthose drawn by Magellan depict geographic domains.

A “location” is a spatial domain. Spatial domains can contain otherspatial domains. A spatial domain that contains a second spatial domaincan be said to encompass the second spatial domain. Since some spatialdomains are large or not precisely defined, any degree of overlapbetween the encompassing spatial domain and the encompassed location isconsidered “encompassing.” Since a spatial domain is a set of elementsfrom a metric vector space, the word “encompassing” means that thelogical intersection of the sets of elements represented by the twospatial domains in question is itself a non-empty set of elements.Often, “encompassing” means that all of the elements in the secondspatial domain are also elements in the encompassing domain. Forexample, a polygon describing the city of Cambridge is a location in thespatial domain typically used to represent the state of Massachusetts.Similarly, a three-dimensional polyhedron describing a building inCambridge is a location in the spatial domain defined by the polygon ofCambridge. The word “location” is a common parlance synonym for a“spatial domain.”

A “location reference” is a sub-media object of a document that a humancan interpret as referring to a location. For example, a sub-string of adocument may be “Cambridge, Mass.,” which a human can interpret asreferring to an entity with representative coordinateslongitude-latitude coordinates (−71.1061, 42.375).

A “geospatial reference” is a location reference to a location within ageographic domain.

“Location-related content” is information that can be interpreted asidentifying or referring to a location within a spatial domain.Location-related content can be associated with a media object in manyways. For example, location-related content may be contained inside themedia object itself as location references, such as names of places,explicit latitude-longitude coordinates, identification numbers ofobjects or facilities or buildings. For another example,location-related content may be associated with a media object by asystem that associates a reference to a media object withlocation-related content that is separate from the media object itself.Such a system might be a database containing a table with a URL fieldand a latitude-longitude field in a table. To obtain location-relatedcontent associated with a media object, a person or computer programmight pass the media object to a geoparsing engine to extractlocation-related content contained inside the media object, or it mightutilize a system that maintains associations between references to mediaobjects and location-related content. The fact that a creator of a mediaobject once lived in a particular place is a piece of location-relatedcontent associated with the media object. Other examples of suchauxiliary location-related content are the locations of physical copiesof the media object and locations of people interested in the mediaobject.

A “sub-media object that is not a location-related content” is asub-media object that is not a location reference. For example, afragment of a text document that says “Eat great pizza in” is notlocation-related content even though the subsequent string may be alocation reference.

A “spatial relationship” is information that can be interpreted asidentifying or referring to a geometric arrangement, ordering, or otherpattern associated with a set of locations. For example, “the alienstraveled from Qidmore Downs to Estheral Hill,” describes a spatialrelationship that organizes the location references “Qidmore Downs” and“Estheral Hill” into an ordering.

A “reference to a media object” is a means of identifying a media objectwithout necessarily providing the media object itself. For example, aURL is a reference to a media object. For another example, media objecttitle, author, and other bibliographic information that permits uniqueidentification of the media object is a reference to that media object.

An “image” is a media object composed of a two-dimensional orthree-dimensional array of pixels that a human can visually observe. Animage is a multi-dimensional representation of information. Theinformation could come from a great variety of sources and may describea wide range of phenomena. Pixels may be black/white, various shades ofgay, or colored. Often a three-dimensional pixel is called a “voxel.” Animage may be animated, which effectively introduces a fourth dimension.An animated image can be presented to a human as a sequence of two- orthree-dimensional images. A three-dimensional image can be presented toa human using a variety of techniques, such as a projection fromthree-dimensions into two-dimensions or a hologram or a physicalsculpture. Typically, computers present two-dimensional images oncomputer monitors, however, some human-computer interfaces presentthree-dimensional images. Since an image is a multi-dimensionalrepresentation of information, it implies the existence of a metric onthe information. Even if the original information appears to not have ametric, by representing the information in an image, the process ofcreating the image gives the information a metric. The metric can bededuced by counting the number of pixels separating any two pixels inthe image. If the image is animated, then the distance between pixels intwo separate time slices includes a component from the duration of timethat elapses between showing the two time slices to the human.Typically, a Euclidean metric is used to measure the distance betweenpixels in an image, however other metrics may be used. Since images canbe interpreted as having a metric for measuring the distance betweenpixels, they are representations of domains. Typically, images arerepresentations of spatial domains. An image of a spatial domain that isassociated with the planet Earth is typically called a “geographic map.”An image of another spatial domain may also be called a “map,” but it isa map of a different type of space. For example, an image showing thefictional location known as “Middle Earth” described in the novels byTolkien is a type of map, however the locations and domains displayed insuch a map are not locations on planet Earth. Similarly, one may viewimages showing locations on the planet Mars, or locations in stores inthe city of Paris, or locations of network hubs in the metric spacedefined by the distances between router connections on the Internet, orlocations of organs in the anatomy of the fish known as a Large-MouthBass. An image depicting a spatial domain allows a person to observe thespatial relationships between locations, such as which locations arecontained within others and which are adjacent to each other. A subsetof pixels inside of an image is also an image. Call such a subset ofpixels a “sub-image”. In addition to simply depicting the relationshipsbetween locations, an image may also show conceptual relationshipsbetween entities in the metric space and other entities that are notpart of that metric space. For example, an image might indicate whichpeople own which buildings by showing the locations of buildingsarranged in their relative positions within a domain of a geographicmetric space and also showing sub-images that depict faces of people whoown those buildings. Other sub-images may be textual labels oriconography that evokes recognition in the human viewer.

A “map image” is an image in which one or more sub-images depictlocations from a spatial domain. A “geographic map image” is a map imagein which the spatial domain is a geographic space.

A “search engine” is a computer program that accepts a request from ahuman or from another computer program and responding with a list ofreferences to media objects that the search engine deems relevant to therequest. Another name for a request to search engine is “search query”or simply a “query.” Common examples of search engines include:

-   -   free-text search engines that display lists of text fragments        from media objects known as “web pages;”    -   image search engines that accept free-text or other types of        queries from users and present sets of summaries of images, also        known as “image thumbnails;”    -   commerce sites that allow users to navigate amongst a selection        of product categories and attributes to retrieve listings of        products; and    -   online book stores that allow users to input search criteria in        order to find books that match their interests. Frequently, a        result set from a book search engine will contain just one        result with several different types of summaries about the one        book presented in the result list of length one. Related books        are often described on pages that are accessible via a        hyperlink; clicking such a hyperlink constructs a new query to        the book search engine, which responds by generating a new page        describing the new set of results requested by the user.

A “search result listing” is the list of references provided by a searchengine.

A “text search engine” is a search engine that accepts character symbolsas input and responds with a search result listing of references to textdocuments.

A “human-computer interface device” is a hardware device that allows aperson to experience digital media objects using their biologicalsenses.

A “visual display” is a media object presented on a human-computerinterface device that allows a person to see shapes and symbols arrangedby the computer. A visual display is an image presented by a computer.

Computer systems often handle “requests” from users. There are many waysthat a computer system can “receive a request” from a user. A mouseaction or keystroke may constitute a request sent to the computersystem. An automatic process may trigger a request to a computer system.When a user loads a page in a web browser, it causes the browser to senda request to one or more web servers, which receive the request andrespond by sending content to the browser.

A “visual indicator” is a sub-image inside of a visual display thatevokes recognition of a location or spatial relationship represented bythe visual display.

A “marker symbol” is a visual indicator comprised of a sub-imagepositioned on top of the location that it indicates within the spatialdomain represented by the visual display.

An “arrow” is a visual indicator comprised of an image that looks like aline segment with one end of the line segment closer to the locationindicated by the visual indicator and the other end farther away, wherecloser and farther away are determined by a metric that describes thevisual display.

The word “approximate” is often used to describe properties of a visualdisplay. Since a visual display typically cannot depict every singledetailed fact or attribute of entities in a space, it typically leavesout information. This neglect of information leads to the usage of theterm approximate and often impacts the visual appearance of informationin a visual display. For example, a visual indicator that indicates thelocation “Cambridge, Mass.” in a geographic map image of the UnitedStates might simply be a marker symbol positioned on top of some of thepixels that partially cover the location defined by the polygon thatdefines the boundaries between Cambridge and neighboring towns. Themarker symbol might overlap other pixels that are not contained withinCambridge. While this might seem like an error, it is part of theapproximate nature of depicting spatial domains.

A “spatial thumbnail” is a visual display of a summary of a media objectthat presents to a user location-related content or spatialrelationships contained in the media object summarized by the spatialthumbnail.

A “digital spatial thumbnail” is a spatial thumbnail comprised of adigital media object that summarizes a second media object, which mightbe either digital media object or other form of media object.

A “companion map” is a visual display that includes one or more spatialthumbnails and the entire media object summarized by the spatialthumbnail. If a companion map is a sub-summary, then may include onlythe sub-media object and not the entirety of the larger media objectfrom which the sub-media object is excerpted.

An “article mapper application” is a computer program that providescompanion maps for a digital media object.

To “resolve” a location reference is to associate a sub-media objectwith an entity in a metric space, such as a point in a vector space. Forexample, to say that the string “Cambridge, Mass.” means a place withcoordinates (−71.1061, 42.375) is to resolve the meaning of that string.

A “geoparsing engine” is a computer program that accepts digital mediaobjects as input and responds with location-related content extractedfrom the media object and resolved to entities in a metric space. Whilethe name “geoparsing engine” includes the substring “geo”, in principlea geoparsing engine might extract location-related content aboutlocations in non-geographic spatial domains, such as locations withinthe anatomy of an animal or locations with a metric space describing DNAinteractions or protein interactions. Such a system might simply becalled a “parsing engine.”

A “text geoparsing engine” is a geoparsing engine that accepts digitaltext documents as input and responds with location-related contentextracted from the document and resolved to entities in a metric space.

An “automatic spatial thumbnail” is a spatial thumbnail generated by ageoparsing engine without a human manually extracting and resolving allof the location references of the media object summarized by the spatialthumbnail. An automatic spatial thumbnail might be semi-automatic in thesense that a human might edit portions of the spatial thumbnail afterthe geoparsing engine generates an initial version. The geoparsingengine may operate by generating so-called “geotags,” which are one typeof location-related content that uses SGML, XML, or another type ofcompute-readable format to describe locations and spatial relationshipsin a spatial domain, such as a geographic domain. For further details ongeotags, see, e.g., U.S. Provisional Patent Application No. 60/835,690,filed Aug. 4, 2006 and entitled “Geographic Text Search Enhancements,”the entire contents of which are incorporated herein by reference.

An “automatic spatial thumbnail of a text document” is an automaticspatial thumbnail generated by a text geoparsing engine in response to adigital text document.

An “integrated spatial thumbnail” is an integrated summary that includesas one or more spatial thumbnails. An integrated spatial thumbnail mayinclude sub-media objects excerpted from the media object beingsummarized, which illustrate location references that relate to thelocation-related content summarized by the spatial thumbnail. Forexample, an integrated spatial thumbnail that summarizes a PDF filemight show text excerpted from the PDF file and a spatial thumbnail witha geographic map image showing visual indicators on locations describedin the PDF's text. For another example, an integrated spatial thumbnailthat summarizes a movie might show a text transcript of words spoken byactors in the movie and a spatial thumbnail showing the animated path oftwo of the movie's protagonists through a labyrinth described in thefilm.

An “automatic integated spatial thumbnail” is an integated spatialthumbnail in which one or more of the spatial thumbnails is an automaticspatial thumbnail.

A “representation of location-related content” is a visual display ofassociated location-related content. Since location-related contentdescribes domains and spatial relationships in a metric space, arepresentation of that content uses the metric on the metric space toposition visual indicators in the visual display, such that a humanviewing the visual display can understand the relative positions,distances, and spatial relationships described by the location-relatedcontent.

A “web site” is a media object that presents visual displays to peopleby sending signals over a network like the Internet. Typically, a website allows users to navigate between various visual displays presentedby the web site. To facilitate this process of navigating, web sitesprovide a variety of “navigation guides” or listings of linkages betweenpages.

A “web site front page” is a type of navigation guide presented by a website.

DESCRIPTION OF DRAWINGS

FIG. 1 schematically represents a graphical user interface fordisplaying a spatial thumbnail and a document summary to a user.

FIG. 2 is a high level flow diagram of a computer-implemented method ofdisplaying information about a media object.

FIG. 3 schematically represents a graphical user interface fordisplaying an integated summary to a user.

FIG. 4 schematically represents a graphical user interface fordisplaying spatial thumbnails for media objects referenced in searchresults.

FIG. 5 is a high-level flow diagram of a computer-implemented method ofdisplaying information about media objects referenced in search results.

FIG. 6 schematically represents a system for generating spatialthumbnails, executing searches, and generating location-related contentfor digital media objects.

FIG. 7 schematically depicts a graphical user interface for displayinginformation about media objects referenced in search results.

DETAILED DESCRIPTION

Overview

A “spatial thumbnail” is a visual summary of location-related content ina media object, such as a document or video. A spatial thumbnail is animage or other form of media containing indicators of places and spatialrelationships that connote, indicate, or otherwise prompt recognition oflocation-related content associated with a media object. Thelocation-related content may be explicitly referenced in the media, suchas specific place names or spatial coordinates, or it may be implied byassociation, such as organizational associations of the authors orprotagonists in the associated media object, historical events depictedin the associated media object, genre of the associated media object orother attributes that connote location-related content or spatialrelationships. The spatial thumbnail may summarize this location-relatedcontent explicitly on a map image using cartographic symbology or it maydepict an approximate spatial sketch of the location-related content.Alternatively, a spatial thumbnail may present audio or video toilluminate the location-related content of a media object. A spatialthumbnail may be integrated with other summarizing elements to form anintegrated summary. A companion map is similar to a spatial thumbnailexcept that it accompanies the full content of the media object to whichit is associated.

Spatial Thumbnails for Document Summaries

Documents are an important type of media object. Digital documents occurfrequently in modern life, both in work and non-work environments.Spatial thumbnails of documents enable several useful thought patternsand work flows that would be difficult without spatial thumbnails. Forexample, web sites owned by newspaper companies typically organize theirfront page visual display to help their visitors assess which newsstories are more interesting or important than others. Given limiteddisplay space and limited attention span of users, such web sitestypically present very short summaries of the news stories. The successof the news site in serving its content to content consumers and thesuccess of the consumers in finding content they desire is largelydetermined by the quality of the summaries presented. By including aspatial thumbnail as a component in such an integrated summary, a website can give its users a different perspective on the content withoutforcing the user to spend time loading or viewing the associated mediaobject in its entirety.

For example, consider two news stories about fishing licenses in the Bayof Fundy. One may refer to locations along the shore of Maine and theother to Nova Scotia and Grand Mannan Island. While a news paper website may contain both articles, the space allotted to describing the twoarticles might only permit twenty words per article, which is not enoughto describe the content of the story and the list of places referenced.However, displaying, for example, a fifteen word extract of the documentinstead of twenty words, and using the extra space to show a smallspatial thumbnail a geographic map image of the area with markers on thelocations referenced in one or the other article can communicate thislocation-related information without consuming additional space on thefront page.

As another example, consider a document describing the detailed behaviorof birds in some parts of South America. Such a document might be storedin an online archive that charges for access to its documents, and thusonly presents summary information to prospective customers. Without aspatial thumbnail, these would-be customers would not know whether thedocument discusses specific sub-regions of the large region of SouthAmerica that matter to their particular interests. A spatial thumbnailcould simply be a listing of salient place names from the document, orit could be a spatial thumbnail image showing such salient locations.

FIG. 1 schematically illustrates a graphical user interface (GUI) (1)for a “thumbnail” software system that allows users to visualize spatialinformation, in the form of spatial thumbnails, within media objectssuch as digital documents. The GUI includes two “areas” for displayinginformation about a media object to a user: a spatial thumbnail (3)representing spatial information within the media object, and a summary(2) of the media object. The spatial thumbnail (3) graphically displaysan image representing location-related content and/or spatialrelationships described in the media object. More specifically, thespatial thumbnail (3) includes an image of a spatial domain, e.g., ageographical map, and one or more markers (4) on the image thatrepresent locations or spatial relationships described or referenced bythe media object, such as points representing cities, or polylinesrepresenting routes from location to another. The summary (3) is asuccinct representation of the media object, or of a sub-media objectwithin the media object, for example an excerpt from the media object, areview of the media object, or an image of the media object or a portionof the media object. Together, the spatial thumbnail (3) and summary (2)allow users to quickly assess whether the media object is of interest tothem, e.g., whether the media object references geographical locations,and subject matter, of interest. Optionally, the GUI also includes areference (10) to the media object, such as a hyperlink or bibliographiccitation information, which the user can use to access the full mediaobject.

FIG. 2 is a high-level flow diagram of steps the thumbnail softwaresystem takes to display information about a given media object. First,the thumbnail software system receives a request to display informationabout the media object (200). For example, the user may click on ahyperlink referencing a book he is considering purchasing, which invokesthe thumbnail software system, causing it to display a summary andspatial thumbnail of the media object as described in greater detailbelow, and thus helping the user assess whether he wants to buy thebook. Next, the thumbnail software system obtains the summary of themedia object (210). Depending on the particular application and the typeof media object, the thumbnail software system may be associated withalgorithms that generate the summary of the media object when the userrequests information about the media object; or, the summary may havebeen previously generated by a separate system, and simply stored in asummary repository from which the thumbnail software system obtains it.

Next, the thumbnail software system obtains the spatial thumbnail forthe media object (220), based on location-related content describinglocation references or spatial relationships described by or relating tothe media object. In some embodiments, the system does this by firstobtaining location-related content associated with the media object(222), such as coordinates for a location referenced within the mediaobject. If the media object refers to multiple locations, thelocation-related content can represent locations corresponding to someor all of these references. As discussed in greater detail below, thelocation-related content can be generated on the fly, e.g., by softwaresystems associated with or incorporated into the thumbnail softwaresystem, or it can be obtained from a repository of previously generatedlocation-related content for media objects.

The thumbnail software system then generates an image based on thelocation-related content (224). The image represents a spatial domainrelevant to the location-related content, e.g. a colorful digitalbackdrop geographic map image showing a geographic domain encompassinglocation(s) described by the location-related content associated withthe media object. For example, if the location-related contentassociated with the media object represents multiple locations in aselected sub-domain of Japan, appropriate backdrop maps include ageographic map image of the entire country of Japan, or a map of theselected sub-domains of Japan. The system generates the map by obtainingit from a map repository or other map image generating system, andoptionally resizing or otherwise modifying the extent of the domainrepresented by the image so as to appropriately represent a relevantsub-domain containing some or all of the location references describedby the location-related content associated with the media object. Forexample, the system could generate a geographic map image by obtaininginformation from a geographical database that the system then uses torender an appropriate image.

The thumbnail software system then optionally generates visualindicator(s) to represent the location(s) and spatial relationshipsdescribed in the location-related content (226). The thumbnail softwaresystem selects the visual indicators based on the number and kind oflocations and spatial relationships described in the location-relatedcontent. For example, if the location-related content describes multiplecities in a country, the system may indicate the locations of thosecities using circular image symbols superimposed on a geographical mapimage at the approximate locations of those cities. For another example,if the location-related content represents a specific area of a givencity, the system may display a polygon that approximates the shape ofthe city. For another example, if the location-related contentrepresents a route taken along a river and up a road from one town toanother, then the system may display a polyline in the image that itgenerates.

The thumbnail software system then simultaneously displays the spatialthumbnail, and the summary of the media object, to the user (230), e.g.,as illustrated in FIG. 1.

In other embodiments, the thumbnail software system does not generatethe spatial thumbnail, but obtains a previously generated spatialthumbnail from an appropriate repository, e.g., database. If the mediaobjects are digital, and collectively stored in a repository, then thespatial thumbnails can be stored with the media objects in therepository and then obtained for display in the GUI, thus obviating theneed to dynamically generate the spatial thumbnails each time the userwishes to view information about the media objects.

The location-related content associated with the media object, which thethumbnail software system uses to generate the spatial thumbnail for themedia object, may be generated by a “geoparsing” software system thatparses the media object (or information about the media object) and usesappropriate analytical tools to determine what location references andspatial relationships the media object contains or describes. Based onthe locations and spatial relationships that the analysis identifies,the parsing software system then generates a location-related contentthat describes the appropriate information in terms of domains,sub-domains, and relationships between entities in a metric space. Forexample, if the media object references “Cambridge, Mass.,” the parsingsoftware system generates a location-related content representing thegeographic coordinates (−71.10161, 42.375). Or, for example, if themedia object describes that “the aliens traveled from Qidmore Downs toEstheral Hill,” the parsing software system generates a location-relatedcontent representing not only the geographic coordinates of the twolocations, but also information ordering the two locations, andoptionally information regarding a route between the two locations. Forsome media objects, the location-related content includes bothreferences to locations and also spatial relationships.

The parsing software system may associate the generated location-relatedcontent with the media object, e.g., by adding the location-relatedcontent to the media as “metadata,” by storing the location-relatedcontent in a database along with the media object, or by indexing thelocation-related content with a reference to the media object. In otherwords, the parsing software system stores the location-related content,and thus a representation of location(s) referenced by or relating tothe media object, in an indexed database that the thumbnail softwaresystem can later access in order to generate a spatial thumbnail for themedia object. Thus, the location-related content need not be generatedeach time the thumbnail software system generates a spatial thumbnailfor the media object. Alternately, the parsing software system can bedirectly associated with the thumbnail software system, in order togenerate location-related content for media objects as users requestinformation about the media objects. In other words, the parsingsoftware system can be configured to generate location-related contentin response to requests for information about media objects, and to passthe location-related content to the thumbnail software system for use ingenerating spatial thumbnails.

The GeoParser software described in U.S. Pat. No. 7,117,199, the entirecontents of which are incorporated herein by reference, is one exampleof a parsing software system that automatically generateslocation-related content for digital media objects. There are a greatvariety of geoparsing engines. Some simply recognize latitude-longitudecoordinate metadata stored inside of media objects, such as the headersand auxiliary files of GeoTIFF images. Others geoparsing enginesrecognize street addresses contained in specially designated fields indatabases associated with documents. More advanced geoparsing enginesuse natural language processing to recognize when substrings of adocument were intended by the author to refer to specific locations,such as “twenty miles north of al Hamra,” where the phrase “al Hamra”could have meant the color read or it could have meant one of many smalltowns and villages in the Middle East, which the geoparsing engine mustdifferentiate between in order to resolve the location reference to aspecific location in a geographic domain.

Note that the media object itself need not be stored in a databaseaccessible to the thumbnail software system, or even be stored in adigital format. The media objects simply need to have an associatedlocation-related content that the thumbnail software system can use togenerate a thumbnail, and a summary that the thumbnail software systemcan display to the user. The location-related content can beautomatically generated using a geoparsing engine or other appropriateparsing software system, or can be manually generated and digitallyassociated with the document in a database that the thumbnail softwaresystem can access. Thus, the thumbnail software system can generatespatial thumbnails for a wide variety of media objects, such as digitaldocuments, living organisms, paper documents, text documents, rocks,videos, email messages, web pages, slide show presentations,spreadsheets, rendering of equations, and music. If the media object isdigital, e.g., digital text, digital images, graphs, numbers, binarydata, or other signals, an appropriate parsing software system can beused to automatically generate location-related content for the object.

Spatial Thumbnails for Book Summary Pages

A book is a special type of document, because books are often large andthus include many sub-media objects, e.g., chapters, paragraphs, andimages. The aggregate value of the large number of sub-media objectstypically contained in a book makes books a particularly valuable typeof document. Spatial thumbnails of books are particularly valuable typesof spatial thumbnails both because books are often valuable and becausethey contain large amounts of information, which is often difficult tosummarize in a small amount of space in a visual display.

To find information in books, traditional text search engine techniquesare often insufficient, because the sheer quantity of information in abook is typically poorly summarized by the short summaries traditionallyprovided by text search result listings. To remedy this, book searchengines often provide larger “integrated summaries” of a book's content.For example, an integrated summary for a book may show several textualexcerpts from different chapters, an image of the cover of the book orof pages from the book, and text from reviewers describing the book. Bycontaining many summaries, such an integrated summary may cover orrepresent most or even all of the information present in the book.

Such an integrated summary may contain sub-media objects excerpted fromthe book, reviews from people who read the book, images of pages fromthe book, images from the book, and other perspectives on the book.

Spatial thumbnails add useful information to integrated summaries forbooks and other relatively lengthy media objects, by allowing a user toquickly assess location references within the book. For example, aspatial thumbnail that summarizes a book like “Around the World inEighty Days” allows would-be readers of the book to appreciate the fullsignificance of traveling around the entire world, without needing tobrowse multiple sections or summaries of the book.

Similarly, guidebooks about Australia tend to cover different areas ofthe massive land with differing levels of depth. Even if two guidebookswith otherwise similar summaries, a traveler interested primarily in thearea of New South Wales can differentiate between the two if a spatialthumbnail for one of the books shows a much greater density of locationsin that area of interest.

FIG. 3 schematically illustrates a GUI (5) for a thumbnail softwaresystem that allows users to visualize spatial information from a bookvia an integrated summary. The GUI includes a first summary area (6)including first spatial thumbnail (3) associated with a first sub-mediaobject (2) from the book, e.g., a first fragment of text from the book;and a second summary area (7) including a second spatial thumbnail (3′)associated with a second sub-media object (2′) from the book. The firstand second sub-media objects are associated with first and second setsof location-related content. The thumbnail software system obtains thefirst and second spatial thumbnails (3) and (3′), which respectivelyrepresent spatial information associated with the first and secondsub-media objects (2) and (2′), as described in greater detail above.Because the first and second sets of location-related content canrepresent different spatial references, the first and second spatialthumbnails can represent different locations, e.g., have different mapsand/or different markers (4).

GUI (5) can also be used to show summaries and spatial thumbnailsassociated with two separate media objects, e.g., books, and thus allowusers to simultaneously visualize spatial information related todifferent media objects. In this embodiment, the first summary area (6)includes a first spatial thumbnail (3) and a first summary or sub-mediaobject (2), associated with a first media object; and the second summaryarea (7) includes a second spatial thumbnail (3′) and a second summaryor sub-media object (2′), associated with a first media object. The GUIcan also include first and second references (not shown) to the firstand second media objects, e.g., hyperlinks to the media objects,hyperlinks to more detailed summaries of the media objects, orbibliographic information about the media objects, thus allowing theuser to learn more about the media object after viewing its summary.

Spatial Thumbnails for Search Results

Search engines accelerate learning, commerce, and other aspects of dailylife. While the conventional generation of a list of references thatallow people to retrieve media objects is important, a fundamentalcomponent of the usefulness of search engines is the summarizing natureof the search results themselves.

Spatial thumbnails can enhance the presentation of search resultsreferencing media objects with significant location information. Forexample, when using a free text based search engine, a user could entera query for “exotic fruit farms” and receive a listing of millions ofweb pages mentioning those words. With only conventional textualsummaries, the user would be forced to guess which search results referto places that they consider interesting. Including a spatial thumbnailin an integrated summary alongside the textual summary for each documentreferenced in a search result makes it possible for users to detect in asingle glance whether a document contains location references ofinterest to them. This is superior to forcing the user to enter keywordsassociated with places of interest to them, because with the inclusionof spatial thumbnails the user can type a shorter, simpler query stringand still understand the spatial significance of the returned results.

FIG. 4 schematically illustrates a GUI (6) for a thumbnail softwaresystem associated with a search engine that allows users to visualizespatial information associated with search results. The GUI includes atext entry box (8) to accept free text queries from a user, and a“submit” button (8′) that the user can click to pass a query to thesearch engine. Alternatively, the user can simply hit “ENTER” on hiskeyboard to pass a query to the search engine. The search engine returnsthe results to the thumbnail software system, which then obtains aspatial thumbnail for search results associated with media objects thatreference spatial information. FIG. 4 illustrates the result of a freetext query that generates two search results associated with mediaobjects that each references spatial information, e.g., geographicallocations, and thus has a spatial thumbnail. First result area (6) isassociated with a first search result referencing a first media object,and second result area (7) is associated with a second search resultreferencing a second media object. First result area (6) includes asummary of a first media object (2), a reference to the first mediaobject (10), and a first spatial thumbnail (3) representing a spatialreference within the first media object. Second result area (7) includesa summary of a second media object (2′), a reference to the second mediaobject (10′), and a second spatial thumbnail (10′) representing aspatial reference within the second media object. Displaying a spatialthumbnail associated with a conventional search result, e.g., a briefmedia object summary and reference to the media object, allows the userto more quickly assess the relevance of the link to the user's interest.

FIG. 5 is a high-level flow diagram of steps that a thumbnail softwaresystem, associated with a search engine, uses to display search resultsand spatial thumbnails associated with the search results. First, thethumbnail software system receives a query from a user (500), e.g., viaa free-text query box presented to the user. The thumbnail softwaresystem then transmits the query to a search engine, and obtains theresults from the search engine (510). When the search engine receivesthe query from the thumbnail software system, it generates a list ofsearch results that reference media objects, e.g., media objects thatexist in a database with which the search engine is in communication, oron the World Wide Web. Digital documents and web pages are examples ofsearchable media objects. Each search result includes a reference to themedia object, e.g., a hyperlink and/or bibliographic information thatthe user can use to directly access the media object, and optionallyalso includes a summary of the media object. For example, some searchengines generate or otherwise obtain a summary for media objectsreferenced in search results.

After obtaining the search results from the search engine, the thumbnailsoftware system obtains spatial thumbnails for search resultsreferencing media objects that have spatial references (520), e.g. usingthe systems and methods described above. Briefly, the system firstobtains location-related content associated with the media object (522).The location-related content represents one or more locationscorresponding to one or more spatial references within the media object;if the media object does not have spatial references, it does not haveassociated location-related content. The thumbnail software system thengenerates an image of a spatial domain, e.g., a map, based on thelocation-related content (524), and optionally generates visualindicators representing the location-related content (526). In otherembodiments, the system obtains a spatial thumbnail that was previouslygenerated and associated with the media object, e.g., from a database,as discussed in greater detail above.

The thumbnail software system then displays the search results and thespatial thumbnails obtained for search results referencing media objectshaving spatial references (530), e.g., as illustrated in FIG. 4. It willbe apparent that searches generally result in an arbitrary number ofsearch results, and that not all search results will necessarilyreference spatial information, so the thumbnail software system will notnecessarily obtain a spatial thumbnail for all displayed search results.

FIG. 6 schematically illustrates a system for generating spatialthumbnails, executing searches, and generating location-related contentfor digital media objects. The system includes a thumbnail softwaresubsystem (20), a search engine subsystem (12), a parsing softwaresubsystem (11), and a document repository (13) that stores digital mediaobjects, e.g., a database or the World Wide Web. The thumbnail softwaresubsystem (20) communicates with the search engine subsystem (12) viacommunication link (16), and communicates with parsing softwaresubsystem (11) via communication link (18). The search engine subsystem(12) communicates with document repository (13) via communication link(15). The parsing software subsystem communicates with documentrepository (13) via communication link (14).

The thumbnail software system (20) generates a GUI for accepting queriesfrom a user and displaying to the user search results based on thequeries, and spatial thumbnails associated with at least some of thesearch results, e.g., the GUI illustrated in FIG. 4. The thumbnailsoftware system (20) obtains the content for the GUI by interworkingwith the search engine subsystem (12) and the parsing software system(11). In operation, when the user enters a query into the GUI, thethumbnail software system (20) forwards the query to the search enginesubsystem (12) via communication link (16). The search engine subsystem(12) process the query, generates search results based on the query, andtransmits the search results to the thumbnail software subsystem (20)via communication link (16). The search results each include a referenceto a media object.

Responsive to receiving the search results, the thumbnail softwaresubsystem (20) attempts to obtain a spatial thumbnail for each mediaobject referenced in a search result. The thumbnail software subsystem(20) transmits the references to the media objects referenced in thesearch results to parsing software subsystem (11) via communication link(18). The parsing software subsystem (11) analyzes the media objects,and generates location-related content for media objects as appropriate,e.g., for media objects that reference locations or spatialrelationships. The parsing software subsystem (11) then transmits to thethumbnail software subsystem (20) the location-related content for themedia objects referenced by the search results. Depending on itsconfiguration, the parsing software subsystem (11) might not generatelocation-related content for every media object that reference locationsor spatial relationships, for example if the parsing software subsystem(11) determines that the reference is insufficiently relevant to warrantexpression in a spatial thumbnail; for such media objects, the parsingsoftware subsystem (11) returns a null result to the thumbnail softwaresubsystem (20).

Based on the location-related content obtained from the parsing softwaresubsystem (11), the thumbnail software subsystem (20) obtains spatialthumbnails for media objects associated with location-related content asdescribed in greater detail herein. The thumbnail software system (20)then displays the search results and the obtained spatial thumbnails tothe user via the GUI.

FIG. 7 schematically illustrates a GUI (27) displaying the results of asearch performed for the query “tree farm.” The GUI (27) includes a textentry box (24) containing the query “tree farm” (23), a “submit” button(25), a plurality of search results that include references to mediaobjects (22), e.g., web pages, summaries of the media objects (26), anda plurality of spatial thumbnails (20) adjacent the correspondingsummaries (26) and references (22). As illustrated in FIG. 7, the resultfor “Bent Tree Farms” does not have an associated spatial thumbnail, andthe GUI simply displays a blank space (21) in the area adjacent thesummary (26) and reference (22) for that search result, where a spatialthumbnail would otherwise have been displayed.

The query received from a user may come in several forms. Any type ofuser input that may be transformed into a database query in the commonlyused “Structured Query Language” (“SQL”) could be a query. Examplesinclude, free-text input, selecting one or more options from a list ofoptions offered to the user, and selecting a variety of differentoptions from different options lists and submitting the combined set ofoptions to the computer system as a query. For example, a web site mightoffer several “categories” of content that a user can choose amongst bynavigating to different links presented in a web page. By navigatingthrough a particular sequence of links, the user constructs a querycomprised of the selections made in navigating the links. The resultingquery is then processed by the computer system generating the web page,and a set of search results are presented to the user in the resultingweb page. The page typically contains means of issuing additionalqueries to generate additional result sets. Any of these means ofentering queries can generate search results accompanied by spatialthumbnails for the media objects referenced in the generated searchresults.

Some embodiments utilize a web browser user interface. To displayspatial thumbnails in a search listing, the user interface displayconnects together different parts of the system. For example, a listingof search results may be displayed to a user via a web browser thatdisplays HTML. The HTML may be generated by a script that accepts userqueries via CGI parameters in the URL that requests the HTML. The scriptpasses the user query to a secondary process that generates the searchresult listing in a computer readable form, such as an array of datarecords. The data records typically include several data elements,including a URL to the original media object, a URL to a copy of themedia object that the search engine has cached, a excerpt or fragment orother form of summary about the media object, or possibly a URL or othertype of reference to such a summary of the media object. The scriptprocesses this information to produce an HTML representation that itpasses to the user's web browser. The web browser renders the HTML intoa visual display. CGI is a common gateway interface and is standard inthe art. URL is a universal resource locator and is standard in the art.HTML is the hyper-text markup language and is standard in the art.

The visual display is typically a list of visual arrays arranged in anorderly fashion on the page, such as one item per row or in a grid ofboxes stacked on the screen. And for each row or grid cell, informationfor a specific result is displayed.

Spatial thumbnails can be included in the search result listing bycreating the script such that for every element in the array of searchresults, it includes an HTML IMG tag in the tow or grid cell displayinginformation for that result. An HTML IMG tag is standard in the art andis an instruction to the web browser that causes the browser to load animage from a source URL. The source URL is designated within the HTMLIMG tag. For example, this is an HT ML IMG tag that would instruct a webbrowser to load the image associated with the URL“http://metacarta.com/imagel.png”

<IMG SRC=“http://metacarta.com/imagel.png”/>

By including these in the result listing, the script that arranges thesearch results can request images that may not have been generated yet.This provides an abstraction layer between the script generating thesearch results display and the thumbnail software system. The imagesource URLs included in the result listing could be to images thatalready exist or the system specified by the image source URL maygenerate the images on the fly.

In order to request a spatial thumbnail of a particular media object,the image source URL must somehow specify or encode a reference to themedia object, so the thumbnail software system can interpret the URLrequest and respond with the right spatial thumbnail. One way of doingthis is to pass a URL for the media object in an encoded format insidethe image request URL. Here's an example of passing a URL encoded insideof another URL:

<IMGSRC=“http://metacarta.com/thumbnail?url=http://news.com/articlel.html”/>

The thumbnail software system can then interpret the “url=” parameterwhich contains the value “http://news.com/articlel.html” and generate aspatial thumbnail for that media object. If the thumbnail softwaresystem determines that a particular media object does not havelocation-related content associated with it, or that the locationrelated content associated with it is not appropriate for a particularsituation, it can respond with a blank image. Such a blank image mightbe a transparent image that allows the background color of the searchresult listing's HTML page to show through to the user, or it mightsimply be white.

Often, the URL for requesting the spatial thumbnail will also allow therequestor to specify additional parameters, such as:

-   -   a width and height of the desired image    -   a designation for a given spatial domain, such as “Earth” or        “Mars” so that the thumbnail software system can prepare a map        image of the desired spatial domain    -   a filtering extent, which is a location reference to a location        within a given spatial domain. If a filtering extent is passed,        it instructs the thumbnail system to only display locations from        the location-related content associated with the media object if        those locations are contained within the filtering extent.    -   a selection of which data layers to use in constructing the map        image    -   a particular color to use for blank images, so that they blend        into the HTML page seamlessly.

The value of the blank images is that it allows the script constructingthe search results to ignore the possibility that some of the mediaobjects may not get a spatial thumbnail. The script simply causes thebrowser to request a spatial thumbnail for every result listing, andthen allows the thumbnail system to fill in the space with an invisibleimage if no spatial thumbnail is available for a given media object inthe list.

The user query passed to the script may be of a wide variety of forms.For example, a query may be only free text, or it may be a selectionfrom a fixed list of choices. For example, a site may offer users a listof categories of content in a list. By selecting an item from the list,the user issues a query that retrieves a list of one or more items. Aquery may be a free text query combined with a spatial domain constraintas described in U.S. Pat. No. 7,117,199.

When working with multiple media objects, such as in a search resultlisting that lists references many media objects, it is sometimesappropriate to generate a spatial thumbnail that depictslocation-related content from more than one of the media objects. Forexample, when displaying a list of search results for a user's query forthe word “tree farm” the system can generate a single spatial thumbnailshowing locations from the location-related content associated with eachof the top five documents.

When generating a spatial thumbnail, the location-related content maycontain a large number of locations. By analyzing the media object usingeither human evaluators or automatic algorithms, the thumbnail softwaresystem can decide that a subset of the location-related content is moreimportant or more pertinent to the media object. These locations aresaid to have higher “relevance,” and when generating the spatialthumbnail, the system may choose to only display a subset of thelocation-related content that is higher relevance than other subsets. Byfocusing the thumbnail's display on higher relevance location-relatedcontent, the system can direct the user's attention to the mostimportant aspects of the location-related content. For example, adocument about Japan might also mention Moscow, but only tangentially.By analyzing the relevance of the different locations referenced in thedocument, the system may decide to show a map image that only coversJapan and excludes Moscow.

The software systems described herein can be run on conventionalcomputers, including memory, storage media, input devices, processors,display devices, and the like.

A number of embodiments of the invention have been described.Nevertheless, it will be understood that various modifications may bemade without departing from the spirit and scope of the invention.Accordingly, other embodiments are within the following claims.

1. A method comprising: obtaining one or more media objects havinglocation-related content in response to a search query, wherein thelocation-related content represents one or more locations correspondingto one or more spatial references in the one or more media objects;obtaining respective spatial thumbnails for the one or more spatialreferences; obtaining respective summaries of the one or more mediaobjects; obtaining a respective one or more hyperlinks referencing tothe one or more media objects; and causing to display for the one ormore media objects the respective spatial thumbnails, the respectivesummaries, and the respective one or more hyperlinks.
 2. A computersystem, comprising: a processor; and a memory including a computerprogram, the memory and the computer program configured to, with theprocessor, cause the computer system to perform at least the following:obtain one or more media objects having location-related content inresponse to a search query, wherein the location-related contentrepresents one or more locations corresponding to one or more spatialreferences in the one or more media objects; obtain respective spatialthumbnails for the one or more spatial references; obtain respectivesummaries of the one or more media objects; obtain a respective one ormore hyperlinks referencing to the one or more media objects; and causeto display for the one or more media objects the respective spatialthumbnails, the respective summaries, and the respective one or morehyperlinks.
 3. A non-transitory computer-readable storage mediumcarrying one or more sequences of one or more instructions which, whenexecuted by one or more processors, cause an apparatus to at leastperform the following: obtain one or more media objects havinglocation-related content in response to a search query, wherein thelocation-related content represents one or more locations correspondingto one or more spatial references in the one or more media objects;obtain respective spatial thumbnails for the one or more spatialreferences; obtain respective summaries of the one or more mediaobjects; obtain a respective one or more hyperlinks referencing to theone or more media objects; and cause to display for the one or moremedia objects the respective spatial thumbnails, the respectivesummaries, and the respective one or more hyperlinks.